Skip to content

feat: Implement Random walk in py-libp2p #822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 29 commits into from
Aug 20, 2025
Merged

feat: Implement Random walk in py-libp2p #822

merged 29 commits into from
Aug 20, 2025

Conversation

sumanjeet0012
Copy link
Contributor

@sumanjeet0012 sumanjeet0012 commented Aug 10, 2025 •
edited
Loading

What was wrong?

Fixes #788

Random Walk Module was not present in py-libp2p. The py-libp2p library lacked a Random Walk mechanism for its DHT implementation, which is essential for continuous peer discovery and maintaining a healthy routing table in dynamic peer-to-peer networks.

How was it fixed?

Implemented Random Walk module in py-libp2p with the following key components:

1. Core Module Implementation

  • Created libp2p/discovery/random_walk/ directory with:
    • random_walk.py: Main RandomWalkDiscovery class that performs periodic random peer lookups
    • utils.py: Random peer ID generation and lookup utilities
    • __init__.py: Module exports

2. DHT Integration

  • Added random walk functionality to DHT implementation
  • Modified DHT classes to support background peer discovery
  • Integrated with existing routing table refresh mechanisms

3. Key Features

  • Periodically generates random peer IDs and performs FIND_NODE queries
  • Refreshes routing table by discovering new peers and validating existing ones
  • Configurable options including:
    • Walk interval: How frequently random walks occur
    • Concurrency: Number of walks to run in parallel
    • Timeouts: Maximum time for each lookup
  • Prevents routing table staleness in networks with high churn
  • Improves overall DHT performance and peer discovery success rates
  • Brings py-libp2p closer to feature parity with Go-libp2p and JS-libp2p

4. Background Process Management

  • Implements non-blocking background task execution
  • Graceful shutdown and cleanup mechanisms
  • Error handling for failed lookups and network issues

5. Examples and Tests

  • Created comprehensive example demonstrating random walk functionality
  • Added integration tests to verify continuous peer discovery
  • Performance benchmarks for different configuration options

To-Do

  • Clean up commit history
  • Add or update documentation related to these changes
  • Add entry to the release notes

Cute Animal Picture

put a cute animal picture link inside the parentheses

@seetadev
Copy link
Contributor

@sumanjeet0012 : Great progress on the Random Walk module! 🎉

The approach you’ve taken makes the module clean, extensible, and in line with the existing py-libp2p architecture. I especially appreciate:

Alignment with specs – You’ve kept the implementation true to libp2p’s discovery goals, which will help ensure compatibility across implementations.

Incremental progress – You’re clearly testing and refining as you go, which shows a disciplined approach to development.

The random walk module is going to be a great step forward for py-libp2p’s peer discovery capabilities. Once it’s fully integrated, it will enhance the robustness and adaptability of the network layer — something that will benefit many projects using py-libp2p.

Keep up the momentum! 🚀 Your work here is making py-libp2p stronger and more feature-complete, and it’s exciting to see it coming together.

@sumanjeet0012 sumanjeet0012 marked this pull request as ready for review August 13, 2025 05:45
@sumanjeet0012
Copy link
Contributor Author

@seetadev The Random walk module is now working as expected.
I will shortly add some test cases and a screencast video.

This PR is now ready for review.

@seetadev
Copy link
Contributor

@seetadev The Random walk module is now working as expected. I will shortly add some test cases and a screencast video.

This PR is now ready for review.

Excellent progress, @Sumanjeet — great to hear that the Random Walk module is now working as expected. That’s a meaningful milestone since Random Walk is central to keeping peer discovery active and healthy in the network. By ensuring it works correctly, you’ve laid the groundwork for stronger connectivity and more resilient DHT lookups.

It’s also really good to know you’re planning to add test cases and a screencast video. Both will add a lot of value: the tests will help us validate behavior across different scenarios, while the screencast will give everyone in the team a clear demonstration of the feature in action — always useful for reviews and onboarding new contributors.

With the PR now ready for review, we can take a closer look at the implementation details and provide feedback. From your updates so far, the direction looks solid and thoughtful. Really appreciate the steady effort you’ve put into this — it’s helping move the project forward in a big way. Looking forward to reviewing the code and tests once they’re up.

pacrob
pacrob reviewed Aug 19, 2025
View reviewed changes
libp2p/discovery/random_walk/config.py Outdated
# Timing constants (matching go-libp2p)
PEER_PING_TIMEOUT: Final[float] = 10.0 # seconds
REFRESH_QUERY_TIMEOUT: Final[float] = 60.0 # seconds
REFRESH_INTERVAL: Final[float] = 10.0 # 10 seconds for demonstration
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everywhere else, this value is 300, but here it is 10.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was done for fast local testing, Now corrected it to 300 seconds.

pacrob
pacrob reviewed Aug 19, 2025
View reviewed changes
libp2p/discovery/random_walk/rt_refresh_manager.py Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't appear that large portions of the code in this file (start, _main_loop, _do_refresh) are hit during test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some more tests covering these functions.

pacrob
pacrob reviewed Aug 19, 2025
View reviewed changes
tests/discovery/random_walk/test_random_walk.py


@pytest.mark.trio
async def test_perform_random_walk_running(mock_host, dummy_peer_id):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cover case where no peers found returns as expected

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a test case for the scenario where no peers are found, which returns an empty array.

@pacrob
Copy link
Member

pacrob commented Aug 19, 2025

Hey @sumanjeet0012, looks good. Just a few comments. Thanks for getting this together!

@sumanjeet0012
Copy link
Contributor Author

@pacrob Ready for next round of review.

pacrob
pacrob reviewed Aug 20, 2025
View reviewed changes
@pacrob
Copy link
Member

pacrob commented Aug 20, 2025

Found one extra paren in a log comment, remove that and I think it's good to merge.

@pacrob pacrob merged commit 94d695c into libp2p:main Aug 20, 2025
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pacrob pacrob pacrob approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement Random Walk Module for Continuous Peer Discovery in py-libp2p
3 participants
@sumanjeet0012 @seetadev @pacrob